Vehicle vision system

ABSTRACT

Vehicle visual systems are disclosed to produce seamless and uniform surround-view images of the vehicle using a number of Ultra Wide-Angle (UWA) lens cameras and optionally HUD systems. A distributive system architecture wherein individual cameras are capable of performing various image transformations allows a flexible and resource efficient image processing scheme.

This application is a divisional of U.S. application Ser. No.14/742,331, filed Jun. 17, 2015, which is incorporated herein byreference in its entirety.

FIELD

The embodiments described herein relate generally to driver assistvision systems, and particularly to creating surround views using ultrawide-angle lens cameras and HUDs.

BACKGROUND

There has been an explosion of employing visual assist technologies inautomotive application in recent years. Cameras are now common featuresin vehicles to assist drivers in performing their tasks. Furthermore, anintegrated visual system would be essential in advancements ofautonomous vehicles. Accordingly, there is an increasing need to improvevehicle visual systems for seamless display of the surroundings inreal-time; to optimize image analytics and decision making processes;and to minimize time and efforts required for maintenance andcalibration of the visual systems.

Exploiting Ultra Wide-Angle (UWA) lenses, having Field of Views (FOV) ofat least 180°, may be part of a solution for these challenges when theyare complimented with image and video processing capabilities. Apowerful image and video processor could provide fast and flexibleprocessing of the acquired images. Further, it could eliminateinstallment of expensive and bulky optics in the vehicle. Yet further,adaptive transformations may enable the driver or other passengers(collectively referred to as the users) to select areas of interest toview and receive a distortion corrected image in real-time. Theco-pending patent application PCT/US2012/027189 teaches an innovativemethod for unfolding images acquired by Wide-Angle (WA) and UltraWide-Angle lenses. The disclosed techniques are based on specializedtransformations, including a Zero-Content-Loss (ZCL) transformation thatmay be applied to the visually distorted images captured through a UWAlens. Accordingly, a perspective corrected image may be obtained with ahigh degree of accuracy for areas of interest, yet all the informationcontained in the original captured scene is maintained. A localized andcontent adaptive correction may further be applied to selected regionsof interest.

The present invention discloses novel vehicle vision systemarchitectures and methods of implementation based on proprietary imageprocessing means. A full rear and peripheral view input may be providedwith minimum number of cameras, where ZCL imagery guarantees no blindspots for the driver. Optionally and preferably, adding a front cameramay provide a 360° image acquisition capability. Such a multi-camera(also referred to as MC from here on) systems may realize variousinnovations in the automotive industry. For example, the rear-viewand/or side-view mirrors and the task of adjusting them, either manuallyor electrically may be replaced by a minimum number of smart cameras.Alternatively, one could envision operating a mirror-less vehicle whereall mirrors are replaced by displays. The latter would be essential forautonomous driving where decision makings would be done by the machineinstead of the driver. Further, such visual systems may be integratedwith alarm systems, tracking systems, communication systems, etc. tofacilitate more features such as recording and communicating the eventson the road.

In a vision system, equally as important as the capture devices, are thedisplay devices. That may be one or more LCD displays, one or moreprojectors, one or more Head-Up-Displays (HUDs), or some combination ofthem. An issue with LCD displays may be their physical positioning andplane of focus that are necessarily away from the driver's front fieldof view and road focused focal point. This means looking away andchanging focus in order to see the LCD display, which can potentially bevery dangerous, even if the distraction lasts only a few seconds. Forthese reasons, we are seeing rapid growth of HUD displays for vehicles,which present information in the form of a virtual image displayed a fewmeters in front of the driver, avoiding having to look away or losefocus. The image is also appropriately positioned not to obstruct thenormal view. Because HUD displays are based on projection optics, andthe windshield functions in itself as an optical element, this leads tovarious geometric distortions. Just as a processor corrects fordistortion of an UWA lens, the same type of processor can correct fordistortions on the display side for a HUD. These corrections may becombined with any user specific corrections for customization of theirview. Furthermore, analogous to multiple cameras, multiple displays(HUDs and LCD displays) may be used to present information in novelways, that are more natural to the driver experience. Multiple HUDdisplays may be blended and stitched together to provide large viewdisplays that can be very useful for applications such as augmentedreality for safety, highlighting objects, displaying map information oreven detailed graphics/video content when the vehicle is in park, etc. A360° view discussed above may be presented on a large view HUD made fromtwo or more smaller HUDs in sufficient detail and correct perspective.Combining multiple HUD displays into single novel views also requiresgeometric correction, which can be facilitated by the same processorused for correcting single HUD distortions. The present invention aspart of its vision system includes multiple display systems, functioningindependently or jointly, to present novel sophisticated views to thedriver. All prior art vision based systems have solely focused on thecapture process, with no consideration of the display side.

This invention benefits teachings of the co-pending applicationPCT/US2012/027189; the content of which are incorporated by reference intheir entirety. All methods and transformations may be implemented bysoftware or in real-time using proprietary hardware implementation ofthe transformations, for example as described in U.S. Pat. Nos.7,324,706 and 8,055,070. The system architectures are embodied in avehicle environment but may be applicable to other environments withsimilar video capture and display settings.

SUMMARY

The embodiments described herein provide in one aspect, a vehicle visionsystem comprising:

-   a plurality of Ultra Wide-Angle (UWA) lens cameras mounted on a    plurality of sides of a vehicle, each camera providing a    corresponding UWA feed, wherein each camera comprises a Geometry and    Color processing (GCP) unit that is configured to pre-process the    corresponding UWA feed for a seamless surround-view image    construction; a central logic configured to combine the    pre-processed UWA feeds and output a surround-view image of the    vehicle; and at least one display unit to display at least one of    the surround-view output image, one or more of the UWA feeds, and    one or more of the pre-processed UWA feeds.

The embodiments described herein provide in another aspect a vehiclevision system comprising:

-   a plurality of Ultra Wide-Angle (UWA) lens cameras mounted on a    plurality of sides of a vehicle, each camera providing a    corresponding UWA feed; at least one HUD comprising a geometry and    color processing unit, wherein the at least one HUD is configured to    receive at least one UWA feed and pre-process the at least one UWA    feed for a seamless surround-view image construction; and a central    logic configured to combine the pre-processed UWA feeds and output a    seamless surround-view image of the vehicle;-   wherein at least one of the surround-view output image, one or more    of the UWA feeds, and one or more of the transformed UWA feeds are    projected on a virtual surface in front of windshield of the vehicle    by the at least one HUD.

The embodiments described herein provide in another aspect a method forproducing a seamless surround-view image of a vehicle, said methodcomprising: pre-processing a plurality of Ultra Wide-Angle (UWA) feedscaptured by a corresponding plurality of UWA lens cameras mounted on aplurality of sides of the vehicle, wherein each camera comprises aGeometry and Color Processing (GCP) unit; combining the pre-processedUWA feeds using a central logic to output the seamless surround-viewimage of the vehicle; and displaying at least one of the surround-viewoutput image, one or more of the UWA feeds, and one or more of thepre-processed UWA feeds.

The embodiments described herein provide in another aspect a method fordisplaying a surround-view image of a vehicle, said method comprising:receiving a plurality of Ultra Wide-Angle (UWA) feeds, captured by acorresponding plurality of UWA lens cameras mounted on a plurality ofsides of the vehicle, by at least one Head-UP-Display (HUD) comprising aGeometry and Color Processing (GCP) unit; pre-processing the pluralityof received Ultra Wide-Angle (UWA) feeds for a seamless surround-viewimage construction by the at least one HUD; combining the pre-processedUWA feeds using a central logic to output the surround-view image of thevehicle; and projecting at least one of the surround-view output image,one or more of the UWA feeds, and one or more of the pre-processed UWAfeeds through at least one Head-UP-Display (HUD).

BRIEF DESCRIPTION OF THE DRAWINGS

For a better understanding of the embodiments and/or relatedimplementations described herein and to show more clearly how they maybe carried into effect, reference will now be made, by way of exampleonly, to the accompanying drawings which show at least one exemplaryembodiment and/or related implementation in which:

FIG. 1 illustrates a method, incorporated by reference, for applying azero-content-loss transformation to an image acquired by a UWA lenscamera;

FIG. 2 illustrates a vehicle equipped with a plurality of UWA lens (FOVof 180° in this example) cameras on its sides. The overlapped regions ifthe cameras are also shown;

FIG. 3 illustrates a top-down or bird's-eye view of a vehicle as ifviewed by an imaginary camera above the vehicle and looking down;

FIG. 4 illustrates an example of constructing a stitched-view from aplurality of views captured by a corresponding plurality of cameras.

FIG. 5A illustrates a central system architecture for processingmultiple inputs from a plurality of cameras;

FIG. 5B illustrates a distributive (edge) system architecture forprocessing multiple inputs from a plurality of cameras;

FIGS. 6(A)-6(D) illustrate an example of a top-down view imageconstruction: 6(A) shows four images taken from four UWA lens camerasarranged to capture front, rear, left and right views; 6(B) shows theresult of a geometry transformation applied by each of the four camerasto the respective input image to correct for UWA distortion andperspective; 6(C) shows the result of intensity adjustment applied byeach of the four cameras to the respective perspective corrected images;and 6(D) shows the resulting top-down surround-view obtained by addingthe four images in FIG. 6(C).

FIG. 7 illustrates a distributive (edge) system architecture forprocessing multiple inputs from a plurality of cameras and displayedthrough a corresponding plurality of projectors;

FIG. 8A illustrates a generic architecture for multiple displays thatare capable of communicating with each other and with the central logic;

FIG. 8B illustrates an embodiment of the invention, wherein a systemcomprises a plurality of cameras and displays with distributed (edge)processing capabilities;

FIG. 9 illustrates an exemplary embodiment of the invention where largevirtual images from multiple displays are combined and appear in frontof the windshield of a vehicle;

FIG. 10 illustrates an exemplary system of multiple cameras and displayswith distributed processors that are capable of calibrating fordifferent windshield surfaces and blend together to form a largervirtual display;

FIG. 11 illustrates a system of multiple displays consisting of aplurality of HUDs that project virtual images at different distances anddepths;

FIG. 12 illustrates an exemplary mirror-less vehicle where a combinationof cameras and displays are used to replace the traditional mirrors;

FIG. 13A illustrates an exemplary system where a close up virtual imageallows for multiple users to simultaneously view and possibly interactwith the same content; FIGS. 13B and 13C show examples of a combined,blended and corrected virtual image formed by 3 DLP projectors onto awindshield in a lab environment;

FIG. 14 illustrates an exemplary system of multiple cameras and multipledisplays that may enable augmented reality (AR) and image analytics;

FIG. 15 illustrates an embodied method for correcting for imagedistortions and combining images from multiple sources; and

FIGS. 16(A)-16(E) show a lab setup with 2 DLP projectors and awindshield, where no correction is applied (16A), after applyingcorrection for geometric projection distortions (e.g. keystone) andwindshield distortion (16B), and after applying edge blending to theleft projector (16C) and to the right projector (16D), and finally aftercombining the edge blended portions (16E).

It will be appreciated that for simplicity and clarity of illustration,elements shown in the figures have not necessarily been drawn to scale.For example, the dimensions of some of the elements may be exaggeratedrelative to other elements for clarity. Further, where consideredappropriate, reference numerals may be repeated among the figures toindicate corresponding or analogous elements.

DETAILED DESCRIPTION

It will be appreciated that numerous specific details are set forth inorder to provide a thorough understanding of the exemplary embodimentsdescribed herein.

However, it will be understood by those of ordinary skill in the artthat the embodiments and/or implementations described herein may bepracticed without these specific details. In other instances, well-knownmethods, procedures and components have not been described in detail soas not to obscure the embodiments and/or implementations describedherein. Furthermore, this description is not to be considered aslimiting the scope of the embodiments described herein, but rather todescribe the structure and operation of the various embodiments and/orimplementations described herein.

FIG. 1 illustrates an image processing method as taught in theco-pending disclosure, as incorporated by reference, according to whicha zero-content-loss (ZCL) transformation may be obtained and applied toan UWA lens input image. An image/video is captured by a capture devicethat is equipped with a UWA lens 100. The captured image/video isreferred to as the input image/feed. Digital capture devices(interchangeably referred to as cameras) are normally equipped with animage sensor, such as a CCD or CMOS, through which the image data 110may be obtained. The image data 110 comprise spatial and colorcoordinates of every image pixel of every video frame. In addition, someform of Image and Signal Processor (ISP) is normally embedded in camerasto enable standard functions such as color/brightness/contrastadjustment, white balance, noise reduction, sharpening, etc. Raw UWAimages are known to appear curvy and within an oval (circular being aspecial case) image boundary. This is due to UWA lens mappings, wherebya view from the real world three-dimensional (3D) object space is mappedonto a planer two-dimensional (2D) image space. The lens mappingtransformation needs to be identified 120. It may be either provided bythe lens manufacturer or modeled mathematically. Alternatively, a moreprecise and general mapping may be obtained empirically. Next, a 2Dsurface is selected 130 to fully encompass the FOV of the lens in 3D(i.e. the 2D surface is generally curved). This covering surface is alsoreferred to as the FOV-envelope. A transformation 140 is thenconstructed to map an output image plane onto the FOV-envelope surface,where every pixel is accounted for, and in such a way that the outputimage 170 is substantially improved for perspective while all imagecontent is preserved. Hence the ZCL transformation is achieved byconcatenating the lens mapping and the FOV-envelope transformation 150,and applying the result to the input image 160. The above approach mayalso be applied to a multi-camera system. For a Multi-Camera (MC)system, the FOV-envelope may be seen as any surface that fully orpartially encompasses the FOV of the full system, which is the union ofthe individual camera field of views. The extent of the FOV-envelope isdetermined by the final FOV required for the multi-camera system. Forexample, in a cylindrical surround view, the FOV-envelope will be acylinder surrounding the vehicle with a height on the order of thevehicle height. The FOV-envelope may also be disjoint sections of theindividual camera FOV-envelopes, if that is the desired view. Often thepurpose of multi-camera systems is to stitch views of the differentcameras to produce a much larger FOV (up to 360°) than what is availablefrom a single camera. In this case, a stitching envelope may be defined,which is essentially a particular choice of the FOV-envelope tofacilitate stitching. The stitching envelope can also be seen as atransformation of the FOV-envelope to a surface that is better suitedfor stitching. In this manner, all transformations, single ormulti-camera, stitched or not, can be viewed as constructing theFOV-envelope 130, followed by the transformation 140 that maps thecamera images onto the FOV-envelope, with possible overlapping regionsfor stitching, which correspond to overlapping FOVs of the individualcameras.

Selection of the FOV-envelope and mapping of the output image to thesurface of the FOV-envelope determines the type of view, for example a360° panoramic view or a perspective view, and the areas selected fordisplay. In particular, parts of the view can have differenttransformations. Therefore, if local perspective adjustments to thetransformations 180 are required, one could re-select the envelope 130and start over in order to modify the displayed output 170 accordingly.Alternatively, a different envelope transformation 140 can be appliedwith the same envelope (i.e. no need to alter 130) to tweak the outputimage. The local adjustments can be performed manually via an operatorby selecting one or more areas of interest that need to be corrected, orit could be automated in response to changes in the captured image 180.Additional processing could also be applied to the selected areas, suchas zooming in/out, rotation and reflection operations.

FIG. 2 illustrates an exemplary arrangement of a plurality of cameras ina vehicle visual system. At least one UWA lens camera is mounted on eachof rear side 210, left side 220 and right side 230 of a vehicle 200.Preferably, another camera may be mounted on the front side 240 of thevehicle to enable the visual system with a full surround-viewcapability; where surround-view refers to any view that is based on twoor more cameras, which often cover a 360 view surrounding the cameras.In the example of FIG. 2, FOV of the cameras are depicted as straightdotted lines covering 180°. Inevitably, there would be overlaps betweenFOV of the cameras in the four corner regions 250 surrounding thevehicle. For these regions 250 issues of system calibration, cameraalignment, and image blending would become crucially important in orderto obtain a seamless integrated surround-view. The exact location of thecameras and the spacing between them may be theoretically irrelevant,due to their extensive FOV. However, there are practical considerations.For example, one may choose near the apex of the curvatures on the sidesin order to maximize the unblock portion of the field of view. In thiscase, one may choose to use a UWA with a FOV of slightly larger than180° in order to compensate for the curvatures and leave no blind spots.Side mirrors and bumpers are other convenient spots for encasing thecameras. A desired view would also determine the orientation andpositioning of the cameras. For example, it is more important for adriver to look closer to the curbs and ground rather than looking up inthe sky. Hence, the cameras may preferably be tilted or pointeddownwards. Other factors such as protecting the cameras from weatherelements, collisions, aesthetics, possible hiding of the cameras, etc.should be considered in placing the cameras too. It should be noted thatfor lengthy vehicles, such as trucks, or if a higher image resolution isneeded, more than one camera may be placed on any one side. A skilledperson in the art may extend the principles disclosed in this inventionto any number of cameras. Four is the minimum number of cameras requiredto produce a full surround-view of a vehicle and should not be viewed asa limitation.

Although the arrangement of the cameras shown in FIG. 2 enables viewingof a very large field of view horizontally and vertically (at least 180°up-down and right-left), not all angles are of equal interest. Differentmodes of display may be desired depending on the driving situation. Forexample, when driving forward, a driver would be mainly interested inwhat is on the road, ahead, behind and around her, perhaps more or lesswithin a ±45° above and below her line of sight, consistent with atraditional driver viewing experience. Accordingly, a cylindricalmapping may be applied to the FOV-envelope surface 130 to obtain a 360°panoramic all-around-view mode to be displayed for the driver. Unlikethe traditional window and mirror based viewing, the resulting imagewould be free of obstacles and blind spots.

Alternatively, in another mode of operation, as illustrated in theexample of FIG. 3, a top-down view or so called Bird's-Eye view 300 maybe mapped off the feeds captured from the individual cameras to simulatea surround-view of the vehicle as if captured from a single camera 320above the vehicle and looking down. It should be noted that the line ofsight of the top-down view 340, i.e. orientation of the imaginary camera320, may be arbitrarily chosen and is not limited to the straight downview as shown in FIG. 3. The top-down display mode would be useful, forinstance, during parallel parking, backing up, and driving throughnarrow alleys.

Generally, displaying output of multi-camera systems can be categorizedas either stitched view or non-stitched view. In a stitched view, two ormore cameras output are combined to generate a view from the perspectiveof a virtual camera placed in an arbitrary location. Overlapping betweenthe cameras FOVs is utilized to stitch individual camera feeds.Panoramic and Bird's-eye views, discussed above, are two examples ofstitched views. Furthermore, relative orientations of the cameras arerequired to generate stitched views that can be obtained throughgeometry calibration.

Example of FIG. 4 illustrates the steps and concepts involved ingenerating a stitched-view as if viewed by a virtual camera 410 that isplaced at a desired location. The concept of FOV-envelop, mentionedabove, naturally extends to a multi-camera system. Here instead ofspanning the field of view 450 of an actual camera 460, FOV-envelop 420would cover the full or part of the field of view of the virtual camera410. Since the camera 410 is virtual, the field of view spanned by theFOV-envelop 420 could be as large as 360° and may only be limited by thephysical placements of actual cameras, their field of views; and/or bythe mapping from FOV-envelop to the output. In addition to theFOV-envelop 420, in multi-camera systems, there is also astitching-envelop 430. The stitching-envelop 430 is a 2D manifold (e.g.a cylindrical surface in this example) where the FOV-envelop 420 isoutward mapped to facilitate stitching of content from multiple cameras.Outward mapping onto stitching-envelop 430 may also be guided by the 3Ddepth information mapping and because of that it may also be a timevarying envelop. However, depth information is not always available andtherefore the 3D structure of the scene 440 may not be accuratelypredicted. In such cases, a stitching-envelop is chosen so that itstopology closely imitates the assumed scene. One can easily see thatthese two envelops may easily be integrated into a single envelop. Next,for a given region in the stitching-envelop, the camera 460 thatcorresponds to that region is identified and the stitching-envelop (or aportion that is relevant to that particular camera) is rotated andtranslated to bring the stitching-envelop into the selected camera'sframe of reference. Finally, with the knowledge of camera forwardmapping model, the stitching-envelop is identified with thecorresponding pixels in the individual camera output.

In Bird's Eye view mode, the stitching-envelop may be a plane lying onthe ground and the FOV-envelop is a perspective plane of the virtualcamera. As the driving surface is mostly flat, the topology of thestitching-plane agrees with it. However, when a 3D object appears on theground and violates the assumption, it causes perspective stretchingdistortion. To achieve accurate stitching, one also requires obtainingthe orientation and placement of the ground relative to the virtualcamera. This can be obtained as part of the extrinsic calibration of theBird's Eye.

In 360°-panoramic view mode, the output is mapped to a cylinderFOV-envelop which is subsequently transformed to another cylindricalstitching-envelop with a bigger radius. Translation between the camerascauses parallax effect in the stitching-envelop. This can be minimizedreasonably, for example, by making the radius of the stitching-cylinderconsiderably larger than the maximum distance among the cameras.

There are a number of challenges in creating a surround-view image fromthe plurality of input images. First, the plurality of cameras 210, 220,230, and 240 should be aligned with one another in such a way thatobjects moving from one camera view to another retain their integrityand shape. One advantage of using UWA lenses is, due to their very largeFOV, mechanical alignments may be eliminated. Instead an electroniccalibration scheme may be performed once or every once in a while toassure seamlessness and integrity of the resulting output images.Second, the portions of the image extracted from the correspondingcameras generally vary in brightness, sharpness, contrast, and othervisual features. A seamless combinatory image requires individualadjustment of color and/or brightness for each camera to make the outputimage uniform. Third, the image portions should be processed for aseamless blend in the overlapped areas 250, regardless of what imageryscheme is used for the visual system. Various architectures may bedesigned to achieve a surround-view image construction.

FIG. 5A conceptually illustrates the more traditional centralized systemarchitecture where a plurality of cameras, e.g. four cameras 510, 530,570 and 590 in this example, are mounted on four sides of a vehicle.Each camera provides a corresponding feed to a central processor 500. Inthis configuration the cameras do not need to have more than a basicImage and Signal Processor (ISP), having a limited functionality as faras their image processing capabilities are concerned. In some cases, theISP may even not be included, instead raw data from the image sensorcould be sent to the central processor that will implement all the ISPfunctions. The feed from each of the cameras are collected by thecentral processor 500. In such a central configuration, the centralprocessor 500 must include a powerful Geometry and Color Processing(GCP) unit, and possibly even a multi-channel ISP. The central processor500 collects raw feeds from the individual cameras and performs all thenecessary geometry/color transformations, UWA lens image mapping,perspective correction, brightness adjustments, and surround-view imageconstruction. Additionally, multiple input feeds may be processedseparately by the central processor 500 for individual display. Theproprietary of the centralized systems would therefore mainly be in thecentral processor 500. Centralized system architecture may be found invarious forms in the prior art. It should be noted that the centralprocessor 500 may be more than one physical processing core or evenseveral processing chips, with the distinguishing trait being theprocessing elements would all be at a distance from the camera, normallyby the dashboard controls of the vehicle. The main advantage of thisarchitecture is that the cameras could be simple and inexpensive,particularly useful when greater number of cameras exists in the system.However, since the cameras have a limited or no ISP functionality, thebulk of tasks required for processing and preparing a finalsurround-view image must be carried out by the central processing unit500. Most importantly, the individual cameras in this configuration arenot capable of de-warping UWA lens raw feeds and performing extremegeometry transformations demanded by users. In order to synthesize asurround-view image, a portion of the final image provided by thecorresponding camera should be selected, as per the central processor'scommand. The portions may be processed and stitched together by thecentral processor 500 to obtain the final output image. Therefore, thecentral processor 500 would have a heavy processing duty in thisarchitecture. If it fails, there would be no backup and the entirevisual system might shut down. Multiple redundant processors couldprovide the backup and reduce this effect; however this would lead to amore expensive system. Furthermore, for views which may only requirepart of the image, the full image still needs to be sent to the centralprocessor, leading to large amounts of data constantly being sent aroundthe vehicle. This high bandwidth becomes a major issue as cameraresolutions increase.

An alternative architecture is embodied in this invention, an example ofwhich is illustrated in FIG. 5B. Each camera 520, 540, 560 and 580 maybe equipped with its own specialized Geometry and Color Processing (GCP)unit in addition to the ISP, collectively referred to as the edgeprocessor (as opposed to a central processor). The GCP and ISP may bephysically implemented in the same processor or in two separateprocessors. In this arrangement, referred to as the edge processing ordistributive system, the role of a central processing unit may becomeminimal or at most equal to the participating cameras. Hence, it isreferred to as the central logic 550. Accordingly, individual feeds maybe fully pre-processed by the corresponding cameras, either forindependent display or in preparation for a combination. Thepre-processed portions of the image are provided to the central logic550 that may simply combine the individually pre-processed feedspixel-by-pixel. Pre-processing may include one or more of geometrytransformation, UWA lens image mapping, perspective correction andcolor/brightness adjustments. For example, image blending may beachieved by adjusting intensity of the geometrically corrected views insuch a way that the overlapped areas would have the same brightness asthe non-overlapped areas, when added up together. Although the centrallogic 550 may be equipped with a similar geometry and color processingunit, its role in the distributive architecture is primarily adding ofpixel color values, effectively acting as a combiner.

Essentially, each camera in this configuration becomes a smart cameranot only capable of image processing and self calibrating, but alsocapable of serving as a master to control and manage the othercomponents on performing their processing duties, realizing the conceptof edge processing. In the distributive architecture, if one componentmal-functions and goes offline, another component may take its place.Therefore, the image processing burden of the central logic 550 may beminimal. According to another embodiment, the central logic 550 is alsoequipped with the same processing capabilities, in such a way that itwould be able to serve as the master too but in equal status as thesmart cameras. In a sense, that is a democratic architecture where thesmart cameras and optionally the central logic or processing unit may beutilized as a processing master. The cameras may further have directcommunication between each other, as shown by the dotted lines, wherebythey may query one another for information. For example, uniform colorcorrection may be achieved by one camera looking at color statisticsfrom all cameras, and computing an improved set of color settings, whichare then transferred to the respective cameras. Note the physicalconnection may still be a single wire running to a central point fromeach camera, however, the central logic does not need to participate inthe communication.

FIGS. 6(A)-6(D) illustrate an example of an edge processing procedurewhere a 360° surround-view top-down image is created by a distributivesystem in a lab environment. FIG. 6(A) shows four images taken from fourUWA lens cameras arranged as in FIG. 5. The four sides are arbitrarilynamed Front 612, Rear 614, Left 616 and Right 618 views to be consistentwith vehicle terminology of FIG. 2. FIG. 6(B) shows application of anappropriate geometry transformation performed by each camera's GCP unitin such a way that the UWA lens image is mapped for the top-down viewingand the perspective distortion, clearly present in FIG. 6(A), iscorrected. At this stage, the four perspective corrected views 622, 624,626 and 628 are not ready for a pixel-by-pixel addition by the centrallogic 550. That is due to the overlapped areas (250 in FIG. 2) betweenimages of every two neighboring cameras. The GCP units may further applya pre-calculated intensity (luminance) adjustment to the geometricallycorrected images, such as a gradient shown in FIG. 6(C), required forseamless blending of the adjacent views. Finally, FIG. 6(D) shows thetop-down or bird's-eye view 650 obtained by combining the four processedimages 632, 634, 636 and 638 of FIG. 6(C). Since the corrective geometryand color transformations are done a priori, the combination task of thecentral logic 550 could be merely adding pixel color values, e.g. usingaddition logic.

The display modes may be selected manually by the users. In oneembodiment, the vision system further comprises a user interactivemedium, so that the driver could select a view as desired. The displaymodes may also be selected automatically, typically triggered by afunction that may signal a change in driving conditions. For example, ifa right/left signal is activated, the perspective corrected feeds fromthe right/left cameras (626 and 628) may be displayed on a singledisplay unit or brought to the center of a multi-display panel. When thegear is changed to reverse, a rear view 624 and/or a bird's-eye view 650would be more appropriate and may be displayed on a single display unitor brought to the center of multi-display panel automatically. In adistributive approach, the central logic 550 can be a low cost componentas opposed to a complicated multi-channel DSP type image processor.Furthermore, the GCP at the edge is a minimal addition to the camera,when using low power and low cost solutions, for instance, as availablefrom Geo Semiconductor Inc. and described for instance in the U.S. Pat.Nos. 7,324,706 and 8,055,070.

Equally as important in a visual system is the display component(s). Allreal-time visual systems complement their cameras with a component fordisplaying the constructed surround-view, and optionally individual rawor processed feeds from the cameras. In one example, suggested in priorart, a corrected side view and/or rear view could be displayed on oraugmented to the respective side and rear view mirrors. In thisapproach, a reflective mirror image may be replaced, augmented oralternated with a video stream. The mirrors would not be eliminated, butthe images appear in their usual places familiar to the driver. Theadvantage is that a wider and unobstructed view would be available tothe driver. Alternatively, the output image, and optionally the rawinput feeds, could be displayed on the vehicle dashboard. LCD/LEDdisplays have been used in prior art. It is also possible to project theoutput image, e.g. using a DLP projector, on a viewing surface. Allthese methods of displays may be embodied to the visual systemarchitectures described herein.

Moreover, many vehicles are nowadays equipped with a Head-Up-Display(HUD) unit. A HUD normally comprises one or more projectors that projecta virtual image for the driver's view that appears in front of thewindshield, in order to reduce distraction of the drivers and keep theirfocus on the road ahead. Unlike a real image, a virtual image cannot beformed on a screen. It is only viewable by the driver or otherpassengers in the vehicle or may be captured by a camera. Traditionally,HUDs have been used to display driving related information such asodometer, engine rpm, gas level, etc. However, they can also beconfigured to display the pre or post processed views streamed from aplurality of cameras attached to the vehicles. Various configurationsand implementation of multiple displays in a vehicle vision system willbe described next.

In one exemplary embodiment of the invention, as illustrated in FIG. 7,a special HUD system 700 comprising a plurality of projection elementsmay be employed to display the output, and optionally the input, imagescaptured by a plurality of cameras. The input feed from each camera 720,740, 760 and 780 is displayed via a corresponding projector 710, 730,770 and 790. The multiple projection elements may be mechanicallypackaged to form a single functional HUD system 700, or the projectionelements may be individually packaged into separate HUD systems. In thisarchitecture, the cameras do not need to have a GCP unit. Rather, thegeometry and color processing may be performed by GCP units in theprojectors. In a HUD, or any projection based system, geometric andcolor correction also needs to be performed. These corrections areapplied for various optical elements in the light path, including thewindshield, as well as for mechanical placement (e.g. keystone effect)of the projection units. One or more GCP unit in the HUD may correct forgeometry and color distortions of both the cameras and the HUD.

Multiple types of displays may be used in combination with each other tobest deliver the information needed to the driver in the context andformat best suited for the situation. FIG. 8A shows a genericarchitecture for multiple displays 810-1 . . . 810-N all capable ofcommunicating with each other as well as the central logic 850 in thevehicle system via a communication network 820. The communicationnetwork may be as simple as a wired or wireless connection, such asindustry standard 120 connection or Bluetooth, for passing configurationparameters between the displays. These displays include and are notlimited to LCDs, projection displays, augmented mirror, etc, and arecollectively referred to displays herein in general unless specified.The multiple displays can freely communicate with each other withoutgoing through the central logic 850, and the central logic 850 cancommunicate with all displays.

FIG. 8B illustrates an embodiment of the invention, showing a fullsystem comprising a plurality of cameras 830-1 . . . 830-M and displays810-1 . . . 810-N with distributed (edge) processing capabilities toprovide information to multiple users 860-1 . . . 860-K. The processingcapability includes, but is not limited to, geometry, color, andbrightness processing; and could reside in both the cameras 830-1 . . .830-M and displays 810-1 . . . 810-N, or all within the displays only.The processors in the cameras and displays will allow inter-devicecommunication by forming an ad-hoc communication network 820 in order tocoordinate the processing and display effort between the cameras anddisplays. The division of work could be such that, for example, thecameras 830-1 . . . 830-M will correct for the UWA distortion, and thedisplays 810-1 . . . 810-N will correct for the distortion of thedisplay surface, or both the UWA and display correction could beperformed all by the displays. Through the ad-hoc network 820, thecameras and displays can also leverage each other's processing resourcesto perform video analytics or other intense computation. Multipledisplays of the same type (e.g. all projectors) may be blended togetherto provide a large viewing surface to show the input combined outputfrom the cameras, or other infotainment content to multiple users. Thevideo feeds from the cameras may be shown on any of the displays, andthe driver and the passengers may independently customize the cameraviews and contents for the displays. All the intensive and highbandwidth processing is distributed in this architecture, and thecentral logic 850 is allowed to focus on managing other critical vehicletasks. When there are redundant cameras or displays components, thisarchitecture may also facilitate fault detection and failover in casethere is problem with some of the cameras or displays, throughcommunication via the ad-hoc network 820 formed by the distributedprocessors.

The multiple cameras and multiple displays system in FIG. 8B may also beused to enable fully autonomous self-driving vehicles. For example, thecameras will provide the distortion corrected UWA views required for thevehicle artificial intelligence (AI) to process and navigate the realworld autonomously, while the displays can provide the passengers withentertainment and real time vehicle status information of the operationof the autonomous vehicle. When the multiple displays comprise HUDs thatproject virtual images in front of the driver, this system hasadditional safety benefits, by keeping the driver focused on the roadahead, in the event there is issue with the autonomous AI and requiresfalling back to manual human control.

FIG. 9 shows an exemplary implementation of the invention where largecombined virtual images 920 from multiple displays 900, for example twoor more HUDs, appearing in front of the windshield 950 provideinformation to the driver 940 and passenger 960. In the fail oversituations, the large virtual image from the multiple displays canprovide the driver with appropriate alerts or warnings, and the driverwill have vision of the road condition beyond the virtual image, thusreducing reaction and accommodation time for taking control of thevehicle manually.

FIG. 10 illustrates a system of multiple cameras and displays 1000 withdistributed processors that may correct for different distortions causedby windshield surfaces 1050, and blend individual images together toform a larger virtual display surface 1020. In this configuration, acamera 1010 is interior to the vehicle and seeing the virtual display.For example, the camera 1010 may be integrated with the interior domelight. This system enables the ability to calibrate the displays usingthe images captured by the cameras, or the stored calibration data forfixed locations on the windshield 1050 can be interpolated fordisplaying on different windshield locations. The camera feeds may alsobe used to dynamically adjust the brightness and blending of thecombined virtual image 1020 on the displays based on the luminancevariation of the virtual image. The individual or combined displaysurface 1020 could be made into any shape by rendering the content on a3-dimensional (3D) surface 1060 and is not limited to linear planes. Asan example, FIG. 10 shows a combined envelop in the form of a curvedsurface with the virtual image appearing at equal distance to the vieweras the viewer 1040 pans from left to right. This equal distance viewingsurface has the benefit of minimizing the eye's accommodation time asthe viewer's field of view pans.

FIG. 11 illustrates a system of multiple displays 1100 consisting ofHUDs that project virtual images at different distances and depths togive the effect of a 3D volumetric virtual space with layers ofinformation. This is only possible in a vehicle via multiple projectiontype displays, hence another benefit of having multiple HUDs. Differenttypes of information may be provided at different projected depths orlayers to best suit the situation. For example, the dashboard, console,and less critical driving information of the vehicle may be displayed ona layer near the bottom 1120 or sides of the windshield 1150, and thedriving critical information can be displayed on another layer in theline of sight 1140 of the driver 1160 like that of a typical HUD. Thedifferent depths of the virtual images will require the user's eyes tore-accommodate, which is beneficial in this case so that the lesscritical layer's information is not in focus when the driver isoperating the vehicle. The eye's re-accommodation time and head movementrequired in such system is also less when compared to looking forinformation in a traditional dashboard or console system. The depth andlayout of the layers may also be controlled or changed depending on thedriving mode. For example, when the vehicle is parked the depth of thevirtual images could be reduced to bring a larger viewing surface to theuser 1160.

It is envisioned that a mirror-less vehicle may be achieved by using acombination of cameras and displays as illustrated in FIG. 12. Accordingto one embodiment, the function of the side and rear mirrors arereplaced with cameras feeds, and the distortion corrected feed from thecameras are shown via multiple displays to the driver. The displayscould be LCDs on the dashboard in front of the driver, or HUDs thatproject virtual side and rear video feed in the view of the driver. Thismirror-less system has the benefit that the driver will not have to lookaway from the road to check for traffic when making lane changes, andthe distortion corrected UWA video feeds will capture all the blindspots of the vehicle to provide maximum safety. Additional views likepanoramic or surround views can also be presented to enhance themirror-less driving experience. To facilitate transition to mirror-lessvehicle or to provide a fall back option in case there is problem withany of the video feeds, the distortion corrected UWA video feeds can beaugmented onto existing mirror locations. This system will providedrivers with blind spot free images on each mirror, and the informationis available at the accustomed mirror locations. In the event there isissue with any of the feeds, the UWA feed may be turned off and thedriver will still have the ability to use the mirror in a traditionalmanner to operate the vehicle safely.

It may also be envisioned that a large close up virtual display can beused when the vehicle is parked where multiple users can see thecombined output of multiple displays, where the projected depth is lessthan the depth of a typical HUD's virtual image. FIG. 13A shows anexemplary system where the close up virtual image 1320 allows formultiple users to see the same content and enables simultaneousinteraction and sharing of the displayed information. When the vehicleis parked, the system may be used to display a large map of thevehicle's surrounding, show a movie or provide infotainment to allpassengers in the vehicle. The displays 1300 will correct for thewindshield distortion and the un-collimated virtual image allows for awide viewing angle such that it can be seen by all passengers in thevehicle including those in the rear seats. FIG. 13B shows a combinedvirtual image formed by 3 projection units onto a windshield in a labenvironment where the video feeds from the 3 projection units areblended and corrected for windshield curvature. FIG. 13C shows the casewhere a large map is displayed on the windshield by combining thevirtual images from 3 projection units in a lab environment. The largemap maybe augmented by information related to the landmarks or relevantspots in the map.

FIG. 14 shows a system of multiple cameras and multiple displays thatcan enable augmented reality (AR) and analytics for the driver. Bycombining multiple displays 1400 together to form a large virtualsurface 1420 that covers the majority of the view of the windshield1450, the AR objects 1410 and 1430 can be positioned and associated withreal world objects as the driver looks out the windshield through thevirtual surface. The wide combined virtual surface 1420 enables the ARsystem the ability to highlight and signal critical information that areto the sides of the vehicle's windshield, and is not limited tohighlighting information on the road immediately ahead as a traditionalsingle HUD system. For example, with the wide combined virtual surface,the AR system can mark and highlight a potential hazardous wild animalto the side of the road that maybe attempting to cross in front of themoving vehicle. The UWA distortion corrected feeds from the cameras canalso be shown on the displays to enhance the AR experience. The systemalso has the advantage of decoupling windshield distortion correctionhandled by the displays 1400, from the placement of the AR objects ontothe virtual surface of the displays.

FIG. 15 shows the processing flow for correcting the windshielddistortion for the displays, and stitching together the displays to forma larger combined surface 1500. The flow for stitching multiple displaysis very similar to stitching multiple UWA camera feeds for surroundview, where the captured or perceived content (a real world scene forthe camera or some content for the display) is distorted by the lens'optics or the windshield. Geometric characteristic of lens(es) and thewindshield, whether provided or obtained experimentally should beprovided as data 1510. This data may also be obtained using acalibration procedure, for example as discussed in the U.S. Pat. Nos.8,619,248 and 8,768,094. The overall process 1500 can be broken into twomain steps, the optical distortion compensation 1520, and the luminanceedge blending 1540. An optional content selection step 1530 can be usedto scale or crop the content, as well as to split the content acrossmultiple displays. Lastly, 1550 is merely combining the multipleprojected images that is inherently done by virtue of the overlappinglight envelopes of the multiple projections.

FIG. 16A illustrates a system with two projection units and awindshield, when no correction is applied, resulting in a distortedvirtual image with non-uniform luminance in the overlapping regions.FIG. 16B shows the result after applying correction for windshielddistortion, where the result is a large seamless stitched virtual imagefrom the multiple displays, however without the edge blending in theoverlap. This is analogous to FIG. 6B for cameras, except that thecombining is also visible. FIGS. 16C and 16D respectively illustrateapplication of a smooth edge blending gradient to the left and rightprojectors, in preparation of their combination. Finally, FIG. 16E showsthe result after combining the edge blended portions, which is analogousto the blend applied in FIG. 6C. In FIG. 16E, by virtue of the combinedlight envelopes, one can see that the final result is a large seamlessstitched virtual image from the multiple displays, as would be viewableby a driver.

Implementation of the embodiments of this invention related to visualsystems, in particular those of vehicles, may be enabled in preferablyin a hardware processor that may be integrated with various captureand/or display components of the system. Alternatively, the embodimentsmay be implemented in software and applications for a computer-readablemedium.

Many variations and combinations of the above taught vision systems withother technologies are possible. The vision system my further include adata storage unit, where segments of the input or output images may berecorded, either upon the driver's command or automatically. If combinedwith object detection capabilities, the recording may start prior to apossible collision. If combined with communication means, images may betransmitted to authorities.

While the above description provides examples of the embodiments, itwill be appreciated that some features and/or functions of the describedembodiments are susceptible to modification without departing from thespirit and principles of operation of the described embodiments.Accordingly, what has been described above has been intended to beillustrative of the invention and non-limiting and it will be understoodby persons skilled in the art that other variants and modifications maybe made without departing from the scope of the invention as defined inthe claims appended hereto.

1. A vehicle vision system comprising: a plurality of Ultra Wide-Angle(UWA) lens cameras mounted on a plurality of sides of a vehicle, eachcamera providing a corresponding UWA feed; at least one Head-UP-Display(HUD) comprising a Geometry and Color (GCP) processing unit, wherein theat least one HUD is configured to: receive at least one UWA feed; andpre-process the at least one UWA feed for a seamless surround-view imageconstruction; and a central logic configured to combine thepre-processed UWA feeds and output a surround-view image of the vehicle;wherein at least one of the surround-view output image, one or more ofthe UWA feeds, and one or more of the transformed UWA feeds areprojected on a virtual surface in front of windshield of the vehicle bythe at least one HUD.
 2. The vehicle vision system of claim 1, whereinone HUD is further configured to command other HUDs, the plurality ofcameras and the central logic.
 3. The vehicle vision system of claim 1,wherein the central logic comprises a GCP unit.
 4. The vehicle visionsystem of claim 3, wherein the central logic is further configured tocommand the plurality of cameras and the plurality of HUDs.
 5. Thevehicle vision system of claim 1, wherein each of the plurality ofcameras comprises a GCP unit.
 6. The vehicle vision system of claim 5,wherein one camera from the plurality of cameras is further configuredto command other cameras, the plurality of HUDs and the central logic.7. The vehicle vision system of claim 1, wherein the plurality ofcameras, the at least one HUD, and the central logic comprise acommunication means to communicate with one another via a communicationnetwork.
 8. The vehicle vision system of claim 1, wherein the GCPpre-processing of the UWA feeds comprises one or more of: selecting aviewing perspective in the corresponding UWA feed according to a viewingdisplay mode instruction; applying a first transformation to theselected viewing perspective to correct for perspective distortion ofthe UWA feed; applying a second transformation to the perspectivecorrected UWA feed to correct for windshield distortion and projectiongeometric distortions; and adjusting brightness of the geometric UWAfeeds for seamless blending of overlapped image areas.
 9. The vehiclevision system of claim 8, wherein the display mode is selectedautomatically as triggered by changes in driving conditions.
 10. Thevehicle vision system of claim 8, wherein the display modes is selectedmanually.
 11. The vehicle vision system of claim 8, wherein the displaymode is a top-down view.
 12. The vehicle vision system of claim 8,wherein the display mode is a panoramic surround-view.
 13. The vehiclevision system of claim 1, wherein the virtual images corresponding todifferent projection units are displayed at different depths relative towindshield.
 14. The vehicle vision system of claim 1, wherein thevehicle is mirror-less.
 15. The vehicle vision system of claim 1,wherein augmented reality is enabled on the virtual display.
 16. Amethod for displaying a surround-view image of a vehicle, said methodcomprising: receiving a plurality of Ultra Wide-Angle (UWA) feeds,captured by a corresponding plurality of UWA lens cameras mounted on aplurality of sides of the vehicle, by at least one Head-UP-Display (HUD)comprising a Geometry and Color Processing (GCP) unit; pre-processingthe plurality of received Ultra Wide-Angle (UWA) feeds for a seamlesssurround-view image construction by the at least one HUD; combining thepre-processed UWA feeds using a central logic to output thesurround-view image of the vehicle; and projecting at least one of thesurround-view output image, one or more of the UWA feeds, and one ormore of the pre-processed UWA feeds through at least one Head-UP-Display(HUD).
 17. The method of claim 16, wherein the plurality of cameras, theat least one HUD, and the central logic comprise a communication meansto communicate with one another via a communication network.
 18. Themethod of claim 16, wherein the at least one HUD is configured tocommand other HUDs, the plurality of cameras and the central logic. 19.The method of claim 16, wherein the central logic comprises a GCP unit.20. The method of claim 19, wherein the central logic is furtherconfigured to command the plurality of cameras and the at least one HUD.21. The method of claim 16, wherein each camera of the plurality ofcameras comprises a GCP unit.
 22. The method system of claim 5, whereinone camera from the plurality of cameras is further configured tocommand other cameras, the at least one HUD and the central logic. 23.The method of claim 16, wherein one camera from the plurality of camerasis configured to command other cameras and the central logic.
 24. Themethod of claim 16, wherein the pre-processing of the UWA feeds isachieved by one or more of: selecting a viewing perspective in thecorrespond UWA feed according to a display mode instruction; applying afirst transformation to the selected viewing perspective to correct forperspective distortion of the UWA feed; applying a second transformationto the perspective corrected UWA feed to correct for windshielddistortion and projection geometric distortions; and adjustingbrightness of the perspective corrected plurality of UWA feeds forseamless blending of overlapped image areas.
 25. The method of claim 24,wherein the display mode is selected automatically as triggered bychanges in driving conditions.
 26. The method of claim 24, wherein thedisplay modes is selected manually.
 27. The method of claim 24, whereinthe display mode is a top-down surround-view.
 28. The method of claim24, wherein the display mode is a panoramic surround-view.